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Introduction 

To  support  NIMA's  goals  of  improving  analysts'  productivity  and  automating  tasks  currently  performed  by 
analysts,  Veridical  Research  and  Design  was  engaged  in  an  effort  to  determine  the  feasibility  of  developing  a 
neurophysiologically  based  methodology  that  identifies  and  highlights  potential  contours  of  interest  in 
overhead  imagery  to  guide  the  image  analyst  (1A)  in  visual  exploitation  tasks.  The  objective  of  this  effort  was 
to  research  and  develop  a  concept,  based  on  established  neurophysiological  and  psychophysical  knowledge 
of  how  the  human  visual  system  segments  scenes  and  detects  perceptual  groupings  of  features,  that  will 
allow  analysts  to  complete  search  tasks  in  a  shorter  period  of  time  and  with  greater  confidence  in  their 
analysis. 

Accurate  and  timely  visual  analysis  of  imagery  is  of  concern  to  government,  industry  and  academic 
researchers  across  a  variety  of  domains  and  applications.  Search  tasks  are  crucial  to  the  missions  of  many 
exploitation  organizations,  yet  are  among  the  most  difficult.  The  volume  of  imagery,  the  relatively  low 
degree  of  intelligence  content  and  the  competing  demands  on  the  analyst's  time  all  contribute  to  a  less  than 
optimal  search  process.  The  basic  problem  in  image  exploitation  is  that  the  reduced  information  contained 
in  still,  monochrome  images  allows  little  or  no  fast  discrimination  between  objects  of  interest  and 
everything  else  in  the  image.  As  a  result,  image  exploitation  is  inefficient  as  the  IA  is  forced  to  scrutinize 
everything.  Under  these  circumstances,  increases  in  efficiency  can  be  achieved  only  with  decreased  report 
confidence.  The  concept  described  here  is  a  methodology  to  identify  areas  of  an  image  that  are  likely  to 
contain  contours  of  interest,  thereby  quickly  guiding  the  LA's  attention  to  high  probability  target  areas  so 
that  the  total  region  that  must  be  scrutinized  is  substantially  reduced.  The  result  is  improved  allocation  of 
attentional  resources  that  increases  analysis  confidence  and  accuracy. 

While  a  variety  of  mathematically  derived  image  processing  algorithms  have  been  developed  to  accomplish 
this  task,  to  date  few  methods  have  been  found  that  exceed  the  visual  abilities  of  human  analysts  in  the 
assessment  and  evaluation  of  imagery,  particularly  of  natural  scenes.  The  novel  approach  to  imagery  analysis 
described  here  is  based  on  the  Neuronal  Synchrony  Model  and  supported  by  neurophysiological  research  of 
contour  perception  in  the  mammalian  visual  system.  Specifically,  it  involves  differentiating  perceptually 
relevant  contours  from  a  noisy  background  using  a  Neuronal  Synchrony  Model  of  the  striate  cortex.  The 
resultant  potential  contours  of  interest  are  then  highlighted  in  a  manner  that  augments  the  analyst's  own 
visual  processing  of  the  image,  thus  decreasing  workload  and  increasing  the  precision  of  image  exploitation 
tasks.  The  output  of  this  effort  is  a  proof-of-concept  demonstration  of  the  effectiveness  of  this  methodology 
in  enhancing  visual  searches  on  representative  imagery.  The  next  subsection  provides  background 
information  on  current  methods  of  imagery  analysis  and  outlines  the  neurophysiological  research  that 
inspires  the  Neuronal  Synchrony  approach  to  imagery  analysis. 

Background 

Image  exploitation  involves  a  variety  of  techniques  and  methods.  A  typical  processing  strategy  often  entails  a 
full  scene  'reconnaissance'  processing  to  locate  particular  regions  of  interest  and  then  an  in-depth  analysis 
of  the  selected  region  to  detect  and  identify  specific  objects  of  interest  (Guindon,  1997)*  One  approach  to 
automating  this  process  requires  seeking  local  salient  contours  in  an  image  and  then  seeking  target  objects 
in  the  image  based  on  detected  contour  groupings.  One  "bf  the  most  difficult  problems,  regarded  as  a 
bottleneck  in  this  bottom-up  process,  is  caused  by  the  fact  that  low-level  feature  extraction  algorithms  often 
produce  less  than  optimal  results.  Application  of  traditional  computer  vision  methods  such  as  edge 
detection  to  extraction  of  information  from  digital  remotely  sensed  images  is  complicated  by  the 
characteristics  of  aerial  and  satellite  imagery,  including  noise  resulting  from  the  acquisition  process, 
variations  in  illumination  and  geometry  due  to  camera  angles,  as  well  as  the  effects  of  shading  (Sowmya  & 
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Trinder,  2000).  Similarly,  conventional  neural  network  approaches  also  tend  not  to  perform  well  on  such 
imagery  because  of  the  high  degree  of  partial  and  noisy  information  (Raghu  &  Yegnanarayana,  1997).  As 
mentioned  previously,  human  analysts  excel  at  the  ability  to  accurately  detect  features  of  interest  in  imagery. 
None  of  the  current  mathematical  edge  detectors  have  psychophysical  correlates  in  a  manner,  for  example, 
that  the  CIE  L*a*b*  color  model  corresponds  with  human  color  perception  (Wyszecki  &  Stiles,  1982). 
Examination  of  how  the  human  visual  system  accomplishes  these  tasks  at  the  perceptual  and 
neurophysiological  levels  can  provide  insights  to  approaches  that  can  be  applied  to  better  addressing  this 
issue. 

Physiological  investigations  in  human  and  other  mammalian  visual  systems  have  shown  that  simple  cells  in 
the  visual  cortex  of  the  brain  are  sensitive  to  intensity  gradients  of  particular  orientations  and  spatial 
frequencies  across  the  visual  field  (Hubei  &l  Wiesel,  1959).  Thus,  one  of  the  earliest  cortical  representations 
of  a  visual  scene  is  based  on  multi-resolution  and  multi-orientational  receptive  fields.  Further,  it  has  also 
been  shown  that  these  receptive  fields  are  well  described  by  Gaussian  modulated  sinusoid,  or  “Gabor,” 
functions  (Marcelja,  1980).  A  mechanism  employing  these  characteristics  could  conceivably  produce  results 
in  visual  perception  tasks  similar  to  that  of  human  performance. 

One  source  of  insights  toward  such  a  mechanism  comes  from  anatomical  and  neurophysiological  studies  of 
processing  in  the  striate  cortex.  Specifically,  Gilbert,  Das,  Ito,  Kapidia  &l  Westheimer  (1996)  have  proposed 
that  cortical  cells  integrate  information  over  large  portions  of  the  visual  field  through  long-range  horizontal 
connections  and  that  distant  surrounding  features  am  thereby  modulate  the  response  of  these  cells.  In 
addition,  it  has  been  shown  that  these  same  cortical  cells  are  involved  in  neuronal  temporal  synchronization 
(Singer  &  Gray,  1995).  Temporal  synchronization  holds  that  signals  of  neurons  that  are  to  be  grouped 
together  are  correlated  in  time.  Signals  are  evaluated  by  determining  the  temporal  correlations  of  signal 
fluctuations  occurring  over  a  particular  time  scale.  A  model  that  embodies  units  with  characteristics  of 
cortical  cells  in  a  network  of  long-range  connections  and  capable  of  temporal  synchronization  could  well 
capture  many  of  the  characteristics  of  human  contour  perception  (Yen  &  Finkel,  1998). 

Several  models  have  been  developed  that  express  visual  segmentation  in  terms  of  neuronal  firing 
correlations  (Spoms,  Tononi,  &  Edelman,  1991;  von  der  Malsburg  &  Buhmann,  1992;  Yen  &  Finkel, 
1998).  These  models  employ  permanent  excitatory  connections  between  neurons  to  encode  the  likelihood 
of  being  activated  by  the  same  figure,  reflecting  the  Gestalt  laws  of  perceptual  grouping.  Examples  of  these 
connection  types  include  the  connection  between  two  neurons  in  different  locations  in  the  visual  field  that 
correspond  to  the  same  local  feature  (e.g.  intensity,  orientation)  or  excitation  between  neurons 
corresponding  to  the  same  location  in  input  space.  By  synchronizing  neuronal  activity  in  time,  the 
combined  action  of  all  excitatory  connections  binds  neurons  together  that  are  activated  by  the  same  figure. 
In  a  similar  manner,  inhibitory  connections  suppress  simultaneous  activity  in  cells  belonging  to  figure  and 
ground,  making  those  signal  correlations  unambiguous  and  reducing  accidental  coincidences.  The  result  is 
that  the  different  excitatory  connections  create  global  correlations  within  a  figure  or  background  and 
provide  a  basis  for  feature  combinations  required  for  scene  segmentation  (von  der  Malsburg,  1999). 

There  is  a  growing  body  of  evidence  supporting  the  efficacy  of  the  temporal  correlation  hypothesis  as  a 
model  of  visual  feature  integration,  both  neurophysiologically  (Schmidt,  Goebel,  Lowel,  &  Singer,  1997; 
Singer,  1999)  and  computationally  (Deco  &  Zihl,  2000;  Spoms  et  al.,  1991).  Unlike  traditional  neural  net 
approaches  that  have  a  fundamental  combinatorial  coding  problem  in  dealing  with  the  issue  of  feature 
binding  (von  der  Malsburg,  1999),  models  based  on  temporally  correlated  cell  assemblies  circumvent  this 
difficulty  and  accurately  reflect  human  performance  (Hummel  &  Biederman,  1992).  Taken  together,  these 
findings  suggest  that  a  model  employing  a  mechanism  based  on  temporal  correlation  might  perform  at 
levels  similar  to  that  of  the  human  visual  system  and  do  a  better  job  at  identifying  contours  of  interest  than 
traditional  image  analysis  algorithms.  The  next  section  details  our  approach  for  employing  this  model  for 
guided  imagery  exploitation. 
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Technical  Approach 

Objectives 

The  primary  objective  of  this  effort  was  to  develop  a  neurophysiologically  based  model  that  characterizes  the 
perceptually  relevant  contours  of  interest  in  overhead  imagery  of  natural  scenes  in  a  manner  consistent  with 
human  performance,  utilizing  only  information  available  in  the  image.  The  goal  was  to  apply  the  model  to 
highlight  contours  of  potential  interest  in  a  manner  that  augments  the  analyst’s  own  visual  processing.  The 
output  of  this  work  is  a  proofioficoncept  demonstration  for  enhancing  the  analyst’s  ability  to  perform 
searches  on  representative  imagery. 

The  specific  technical  objectives  of  Phase  I  include: 

•  Neuronal  Synchrony  Model  Development 

Development  of  an  initial  Neuronal  Synchrony  Model  of  contour  detection  and  creation 
of  test  imagery 

•  Model  Evaluation 

Evaluation  of  the  performance  of  the  model  and  determination  of  its  effectiveness  in 
supporting  a  more  effective  visual  search  process 

•  Model  Refinement 

Application  of  the  model  to  a  larger  set  of  representative  test  imagery  and  updating  and 
tuning  performance  based  on  the  results. 


The  sections  below  describe  the  overall  methodology  and  results  of  the  effort  to  create  a  neuronal 
synchrony  model  of  image  analysis.  The  information  below  provides  an  overall  summary  of  the  findings  as 
w7ell  as  providing  suggestions  for  further  research.  For  detailed  information  regarding  results  of  preliminary 
efforts  on  these  tasks,  see  Marchak  (2002). 

Task  1  -  Neuronal  Synchrony  Model  Development 

The  primary  goal  of  this  task  was  to  develop  an  initial  Neuronal  Synchrony  Model,  based  on  the  work  of 
Yen  &  Finkel  (1998).  The  model  determines  contours  of  perceptual  relevance  in  imagery  and  outputs  the 
results  for  use  in  highlighting  these  regions  for  subsequent  processing  by  an  analyst,  A  secondary  goal  of  this 
task  was  to  create  test  imagery  that  could  be  used  to  evaluate  the  performance  of  the  model  in  the  next  task. 

Yen  and  Finkel  (1998)  have  developed  a  model  -  based  on  temporal  synchronization  -  for  computing  the 
perceptual  salience  of  contours  embedded  in  noisy  images  and  have  shown  the  effectiveness  of  the  approach 
in  accounting  for  a  range  of  basic  psychophysical  and  neurophysiological  phenomena.  In  coordination  with 
Dr,  Charles  Gray  and  Dr.  Shih-Cheng  Yen,  we  have  been  extending  and  expanding  their  methodology  and 
incorporating  it  in  a  system  that  will  use  the  output  of  the  model  to  guide  analysts  to  probable  contours  of 
interest  in  remotely  sensed  imagery. 

Model  Design 

The  basic  premise  of  the  Neural  Synchrony  Model  is  that  perceptual  salience  arises  from  the  temporal 
synchronization  of  cells  with  long-range  horizontal  connections  in  the  striate  cortex.  The  model  developed 
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here  is  composed  of  Gabor-based  units  modeled  on  striate  cortex  cells  that  are  embedded  in  a  network 
made  up  of  long-range  horizontal  connections  and  capable  of  temporal  synchronization  (Yen  &  Finkel, 
1998).  The  implementation  of  the  model  involved  developing  model  cells  based  on  Gabor-based  units, 
creation  of  network  filters  that  implemented  facilitation  and  inhibition  of  the  long-range  horizontal 
connections  between  units*  and  development  of  a  temporal  synchrony  mechanism  to  determine  perceptual 
salience.  Each  of  these  factors  is  discussed  below. 


Model  Cells 

As  outlined  in  our  proposal,  Gabor  wavelet  transforms  were  chosen  to  model  the  responses  of  cortical  cells. 
The  wavelet  transform  provides  a  mathematical  technique  for  hierarchically  decomposing  an  image  into 
multiresolution  components  that  provide  a  description  of  the  coarse,  overall  texture,  as  well  as  details  that 
range  from  broad  to  narrow  (Mallat,  1999).  The  resulting  set  of  coefficients  can  be  used  to  uniquely  define 
an  image  and  determine  specific  features  across  a  variety  of  scales  and  orientations.  Of  particular  interest  is 
the  fact  that  that  the  same  characteristics  of  simple  cells  in  the  mammalian  primary  visual  cortex  -  spatial 
localization,  orientation  sensitivity,  and  band  pass  preference  -  are  comparable  to  the  basis  functions  of 
wavelet  transforms  (Olshausen  &  Field,  1996).  The  wavelets  essentially  play  the  role  of  receptive  fields  with 
the  response  of  the  neuron  captured  by  the  wavelet  coefficients. 

The  response  of  the  Gabor  filters  used  in  the  model  are  represented  as: 

A(x,  y,  6)  =  JJ f(x,  y,  9)I(x,  y)dxdy  (1) 

where  f(x,y,  0)  is  the  kernel  of  the  steerable  Gabor  filter  oriented  at  0  and  I(x,y)  is  the  input  image. 

Network  Filters 

As  described  in  Yen  and  Finkel  (1998),  the  model  cells  are  embedded  in  a  network  of  long-range  horizontal 
connections  that  are  capable  of  temporal  synchronization.  Under  the  direction  of  Dr.  Yen,  we  developed  a 
MATLAB  implementation  of  this  network  using  facilitation  and  inhibition  properties  based  on 
psychophysical  and  neurophysiological  evidence  of  human  perception  ((Field,  Hayes,  &  Hess,  1993;  Gray, 
1999;  Kovacs  &  Julesz,  1993;  Polat  &  Sagi,  1993). 

Facilitation 

Each  model  cell  receives  weighted  inputs  from  other  oriented  cells  in  its  surround  dependent  on  the 
position  and  orientation  of  the  pre-synaptic  cells,  resulting  in  separate  spatial  zones  of  excitation  and 
inhibition.  The  excitatory  connections  are  made  up  of  two  regions,  a  co-axial  region  which  spreads  out 
along  the  orientation  axis  of  the  cell  and  a  trans-axial  region  that  extends  orthogonal  to  the  cell  orientation. 
Figure  1  from  Yen  and  Finkel  (1998)  shows  the  connection  pattern  of  a  horizontally  oriented  cell,  with  the 
orientation  and  length  of  the  lines  representing  the  “preferred”  orientation  and  connection  strength, 
respectively,  as  outlined  below. 
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Figure  1.  Connectivity  Pattern  of  Horizontally  Oriented  Cell 


The  coaxial  connections  are  consist  with  Field’s  et  al.  (1993)  “association  field”  derived  from  results  of 
human  psychophysical  performance  and  are  based  on  the  “cocircular”  connection  scheme  of  Parent  and 
Zucker  (1989).  The  cocircularity  rule  states  that  for  two  points  lying  on  a  circle,  the  average  of  their  tangent 
orientations  equals  the  slope  of  the  line  between  them.  For  a  cell  of  some  orientation  0A  at  location  A, 
there  is  a  “preferred”  orientation  at  location  B,  (j)B,  described  by  this  rule.  Local  orientation  activity 
distribution  at  B  strongly  facilitates  0A  when  4>B  peaks,  and  decreases  as  the  orientation  of  B  deviates  from 
<^b-  Given  a  post-synaptic  cell  at  orientation  0,  the  preferred  orientation  <j)  at  position  (i,j)  of  the  pre-synaptic 
cell  \| )  is  described  by: 


0(0,/,/)  =  2  tan 


-1 


(A 

h 


-e 


(2) 


The  connection  weights  peak  at  <j>,  falling  off  as  a  Gaussian  function  of  the  difference  in  the  acute  angle 
( |  |  acute)  between  <|>  and  V|/,  with  the  half-width  at  half-height  a\v  for  the  co-axial  connections  specified  by: 


B(6,  i,j,  y/)  =  G(\y/  -  0(0,  i,  j)\acute ,  CTp  (3) 

In  this  scheme,  connection  weights  decrease  with  increasing  angular  deviation  from  the  cell  orientation, 
which  further  reflects  the  results  of  human  visual  performance  that  show  a  preference  for  lines  that  are 
straight  and  of  low  curvature  (Field  et  al.,  1993).  In  addition,  in  accordance  with  the  psychophysical  results 
of  Polat  and  Sagi  (1993),  connection  weights  decrease  as  a  function  of  distance: 


D(iJ)  =  G(jr7f,a‘ 


(4) 


where  crca  is  the  half-width  at  half-height  of  the  Gaussian  function. 

The  trans-axial  excitatory  connections  extend  orthogonally  from  the  cell  with  a  more  spatially  focused 
pattern,  with  the  strongest  connections  closest  to  the  cell  These  connections  are  calculated  in  manner 
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similar  to  the  co-axial  connections  using  and  Gld.  The  co-axial  and  trans-axiai  connections  compete  so 
that  only  one  set  is  active  at  a  time,  permitting  the  stimuli  to  modulate  the  dominance  of  the  connections. 

As  mentioned  above,  the  fan-out  of  the  connections  is  limited  to  low  curvature  deviations  from  the 
orientation  axis  as  well  as  an  orthogonal  region.  A  further  constraint  is  that  the  horizontal  connections  are 
reciprocal,  based  on  neurophysiological  evidence  (Kisvarday  &t  Eysel,  1992).  These  factors  are  addressed  in 
model  as: 


Uf  tan 
and  tan-1 


'A 


\i) 

(A 

UJ 


■e<Kc 


-y/<K 


Uf  tan" 


(A 

■\ 


and  tan'1  [  - 
0,  otherwise 


-W^±K‘ 


(5) 


where  Kc  is  the  maximum  angular  deviation  of  the  co-axial  connections  and  k'  is  the  maximum  angular 
deviation  of  the  trans-axial  connections.  Reciprocal  fanout  constraints  imposed  by  the  pre-synaptic  cell  are 
represented  by  (-i,-j). 

Taken  together,  these  representations  describe  the  facilitation  for  a  cell  at  orientation  0  located  at  position 
(x,y)  as: 

F(x,  y,  9)  =  JJJ  T {9,  i,  j,  y/)D(i,  j)B(9,  i,  j,  y/)A(x  +  i,y+ j,  y/)dy/didj  (6) 


Inhibition 

The  primary  function  of  inhibition  is  to  differentiate  signal  from  noise  based  on  the  level  of  facilitation. 
Random  elements  in  the  background  may  by  chance  be  optimally  oriented  with  a  contour  element. 
Inhibition  serves  to  prevent  these  elements  from  attaching  to  a  target  contour.  The  facilitation  stage  of  the 
model  extracts  cells  receiving  input  above  the  level  of  noise.  This  stage  implements  a  longer-latency 
inhibition  that  arises  from  outside  the  facilitatory  zones: 


I(x,y,9)  = 


Uf  tan-1 
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Uf  tan 
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0,  otherwise 


-0>-±Kt, 
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The  magnitude  of  the  inhibition  suppresses  any  cell  with  total  support  less  than  a  fixed  threshold  so  that  it 
is  suppresses  only  cells  with  weak  support  but  not  affecting  strongly  facilitated  cells. 

Temporal  Synchronization 

As  described  above,  neurophysiological  evidence  suggests  that  the  same  cortical  cells  interconnected  by  long- 
range  horizontal  connections  are  also  involved  in  temporal  synchronization  (Gray,  1999).  In  the  model,  the 
chattering  cells  are  represented  as  homogenous-coupled  neural  oscillators,  with  the  phase  of  each  oscillator 
modulated  by  the  phase  of  other  coupled  oscillators  (Terman  Sc  Wang,  1995).  Each  oscillator  encodes 
some  feature  of  an  object,  even  just  a  pixel.  Each  object  segment  is  represented  by  a  group  of  oscillators  that 
show  synchrony  -  phase-locking  with  zero  phase  shift  -  with  different  groups  of  oscillators  desynchronized 
from  each  other  representing  different  objects. 

Initial  efforts  involved  use  of  the  neural  oscillator  model  described  in  (Terman  &l  Wang,  1995;  Wang  & 
Terman,  1995).  The  model  is  based  on  locally  excitatory,  globally  inhibitory  oscillator  networks  (LEGION). 
A  single  oscillator,  i,  is  defined  as  a  feedback  loop  between  an  excitatory  unit  xt  and  an  inhibitory  unit  yt: 

=  3x,  -if  +2 -yi+I,+Si+p  (8a) 

at 


=  £[r0-  +  tanh(x,  /  P))~yt\ 


(8b) 


where  It  denotes  external  stimulation  to  the  oscillator,  St  represents  coupling  from  other  oscillators  in  the 
network,  and  p  represents  the  amplitude  of  the  Gaussian  noise  term.  Noise  is  introduced  to  actively 
desynchronize  different  input  patterns. 

As  shown  in  Figure  2,  reproduced  from  Wang  and  Terman  (1995),  the  xmuiicline  of  (8)  is  a  cubic  curve 
and  the  y-nullcine  is  a  sigmoid  curve.  When  I>0,  the  curves  intersect  along  the  middle  branch  of  the  cubic 
and  (8)  is  oscillatory,  with  the  periodic  solution  alternating  between  silent  and  active  phases  of  near  steady- 
state  behavior. 


Figure  2.  Nullcllnes  and  periodic  orbit  of  single  oscillator. 
Reproduced  from  Wang  and  Terman  (1995). 
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The  parameter  y  in  (8b)  controls  the  relative  times  that  the  solution  alternates  between  silent  and  active 
phases  of  steady-state  behavior,  while  P  controls  the  steepness  of  the  sigmoid.  The  coupling  term  Sh  is 
described  by: 

S,  =  £  (?) 

keN(i) 

where 


SK(x,0)  = 


_ 1 _ 

1  +  exp  {-k(x  -  0)} 


(10) 


Wd  describes  the  connection  weight  from  oscillator  k  to  oscillator  I,  and  N(i)  represents  the  set  of  adjacent 
oscillators  that  connect  to  i.  9X  controls  the  threshold  for  which  an  oscillator  can  affect  its  neighbors.  Wz 
represents  the  weight  of  inhibition  from  the  global  inhibitor  z,  with  activity  represented  by 

^  =  #(c= o-z)  (ID 


where  0^=0  when  <  6^  for  every  oscillator,  and  1  if  x,  >  ft,  for  at  least  one  oscillator  u  Therefore, 
once  an  oscillator  is  in  the  active  phase,  it  triggers  the  global  inhibitor,  which  inhibits  the  entire  network  as 
shown  in  (9).  However,  an  active  oscillator  spreads  activation  to  nearest  neighbors  and  onto  further 
neighbors. 


Model  Implementation 

Model  Cells 

A  Gabor-based  algorithm  developed  by  Nestares,  Navarro,  Portilla  &  Tabemero  (1998)  was  chosen  to 
represent  the  model  cells.  This  approach  employs  a  multiresolution  pyramid  using  Gabor  basis  functions. 
These  Gabor  wavelets  are  complex  exponential  functions  modulated  by  Gaussian  functions.  These  wavelets 
can  be  tuned  to  a  continuum  of  spatial  positions,  frequencies  and  orientations.  From  the  basic  Gabor 
functions,  the  complete  set  of  functions  used  for  sampling  the  joint  space-frequency  domain  is  obtained 
using  three  manipulations.  Rotations,  with  a  45°  step,  are  used  to  get  four  orientation  channels  in  the 
frequency  domain.  Stretching  by  a  factor  of  two  to  halve  the  frequency  of  the  sinusoid  permits  sampling  the 
frequency  domain  in  octaves.  Translations  in  horizontal  and  vertical  locations  are  used  to  sample  the  spatial 
domain. 

A  MATLAB  realization  based  on  this  algorithm  was  obtained  and  implemented  in  our  computing 
environment.  Using  four  spatial  frequencies,  the  coefficients  of  the  Gabor  transform  were  obtained 
through  convolution  of  an  image  with  each  one  of  the  filters.  The  distribution  by  octaves  of  the  filters’ 
tuning  frequencies  w?as  implemented  as  a  multiresolution  pyramid.  The  same  set  of  filters  was  applied  to 
low-pass  filtered  and  down  sampled  versions  of  an  original  image.  This  pyramidal  implementation  produces 
a  redundant  but  robust  representation  that  permits  reconstruction  by  adding  together  all  of  the  even  Gabor 
channels  plus  their  residuals.  The  resulting  algorithm  permits  calculation  of  Gabor  wavelet  coefficients  at 
four  different  scales  and  four  different  orientations  that  serve  as  the  cortical  units  in  the  Neuronal 
Synchrony  Model.  In  addition,  the  reconstruction  functions  supported  test  image  creation  as  described  in 
the  Model  Evaluation  section.  Complete  details  of  the  Gabor  decomposition  procedures  can  be  found  in 
(Marchak,  2002). 
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Network 

As  described  in  Yen  and  Finkel  (1998),  the  model  cells  are  embedded  in  a  network  of  long-range  horizontal 
connections  that  are  capable  of  temporal  synchronization.  Under  the  direction  of  Dr.  Yen,  we  developed 
MATLAB  implementations  of  this  network  using  facilitation  and  inhibition  properties  based  on  (6)  and  (7). 

Facilitation  filters  were  developed  for  each  of  the  four  orientations  (0,  45,  90,  135  degrees)  using  long-range 
horizontal  connections  weighted  by  both  the  preferred  angle  and  distance  from  center.  The  preferred  angle, 
<|>,  was  calculated  using  (2).  This  value  is  a  function  of  the  difference  between  0,  the  orientation  of  the  filter 
-  which  represents  the  orientation  of  the  post-synaptic  cell  -  and  i|/,  the  orientation  of  the  input  image  - 
which  represents  the  pre-synaptic  cell.  Next  connection  weights,  B,  which  peak  at  (j)  and  fall  off  as  a 
Gaussian  function  of  the  difference  in  the  acute  angle  between  the  <|>  and  vj/,  were  calculated  using  (3).  For 
the  co-axial  connections,  distance,  acd,  was  set  to  40,  while  the  angle  of  the  co-axial  connections,  ctcv,  was  20 
degrees.  For  the  trans-axial  connections,  a'j,  the  distance  of  the  trans-axial  connections,  was  30  and  a'^,  the 
angle  of  the  trans-axial  connections,  was  20  degrees.  In  a  similar  manner,  distance  weights,  D,  were 
calculated  using  (4).  Lastly,  fan-out  of  the  connections,  T,  which  limits  connections  to  low  curvature 
deviations  from  the  orientation  axis  and  to  a  narrow  region  orthogonal  to  the  orientation  axis,  was 
determined  by  applying  (5).  The  fan-out  of  the  co-axial  connections,  Kc,  was  30  degrees  and  the  fan-out  of 
the  trans-axial  connections,  k',  was  10  depees. 

Facilitation  was  then  determined  by  combining  B,  D,  and  T.  In  this  manner,  a  facilitation  filter  was 
calculated  for  an  input  image  and  filter  orientations  of  0,  45,  90,  and  135  degrees.  These  facilitation  filters 
were  applied  to  the  associated  Gabor  output  for  each  image.  The  resulting  output  contained  a 
representation  of  cell  activity  at  each  pixel  for  a  particular  orientation  based  on  facilitation  of  cells  in  its 
surround. 

Inhibition,  based  on  (6),  was  then  applied  to  the  output  of  facilitation  to  suppress  cells  with  weak 
facilitation.  Inhibition  was  implemented  by  assuming  that  each  element  divides  it  support  equally  among 
cells  that  provide  suprathreshold  pre-synaptic  input.  Any  cell  whose  input  was  less  than  a  fixed  threshold, 
0.5,  was  suppressed.  The  outputs  of  the  facilitation  and  inhibition  processes  for  each  of  the  four 
orientations  were  then  recombined  into  a  single  image.  This  image  was  then  submitted  to  temporal 
synchronization  as  described  below. 

Temporal  Synchronization 

A  two-dimensional  LEGION  oscillator  network  was  implemented  in  MATLAB  based  on  the  equations  in 
Wang  and  Terman  (1995)  described  above  and  as  implemented  in  a  C  program  developed  by  Wang  (1994). 
The  oscillator  network  developed  here  consisted  of  a  1000  x  1000  grid  of  oscillators,  based  on  the  input  size 
of  the  imagery  used  in  the  evaluation  below,  with  a  global  inhibitor  as  defined  in  (8)  -  (11),  with  the 
differential  equations  solved  using  a  fourth-order  Runge-Kutta  method.  Each  oscillator  in  the  simulation 
was  connected  to  its  nearest  neighbor  by  a  constant  weight  set  to  2.5  for  simplicity.  The  remaining 
parameters  were  set  based  on  the  values  used  by  Wang  and  Terman  (1995):  e=0.02,  <|>=3.0,  y=6.0,  (3=0.1, 
k=50,  0=-O.5, 0w=0„=O.l,  p=0.02. 

The  LEGION  implementation  begins  by  calculating  the  connection  weights  for  the  eight  immediate 
neighbors  of  each  oscillator.  Next,  the  initial  phase  of  each  oscillator  is  randomly  initialized.  The  net  input 
for  each  oscillator  is  calculated  over  all  inputs,  the  strongest  link  is  traversed,  and  oscillators  that  jump  into 
synchronization  are  found.  All  pixels  in  a  region  defined  by  an  oscillator  are  the  assigned  the  same 
numerical  value,  beginning  at  1  and  continuing  on  through  the  number  of  regions,  up  to  255. 
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The  output  of  the  LEGION  algorithm  is  a  pixel  map  that  assigns  a  unique  number  to  each  region  defined 
in  the  input  region  defined  by  the  correlated  oscillations.  The  resulting  region  values  were  each  mapped  to 
saturation  levels  of  a  yellow  hue  to  produce  highlighted  regions.  All  other  pixels  were  assigned  to  black.  The 
highlights  from  the  oscillator  output  were  then  overlaid  on  the  original  imagery  to  indicate  perceptually 
salient  regions.  The  next  section  provides  examples  of  this  process  and  discusses  model  evaluation. 


Task  2  -  Neuronal  Synchrony  Model  Evaluation 

The  primary  goal  of  this  task  was  to  evaluate  the  model's  performance,  both  in  terms  of  basic  functionality 
and  overall  effectiveness.  This  was  accomplished  through  functional  evaluation  of  the  individual  model 
components,  as  well  as  by  evaluating  model  performance  in  relation  to  its  ability  to  facilitate  visual  search  in 
imagery. 

Functional  Evaluation 

Each  of  the  separate  components  of  the  Neuronal  Synchrony  Model  was  evaluated  to  determine  that  the 
output  accurately  matched  the  design.  The  individual  elements  of  the  facilitation  filter  -  B,  D,  and  T  -  and 
the  inhibition  component  were  compared  with  manual  calculations  of  a  subset  of  the  inputs.  Any 
discrepancies  were  investigated  and  corrected.  The  LEGION  model  implementation  was  similarly 
evaluated,  first  by  comparing  output  to  artificial  stimuli  containing  separate,  distinct  regions  and  then  by 
application  to  the  imagery  to  be  used  for  human  performance  testing.  The  results  were  accurately 
functioning  modules  that  carried  out  the  calculations  necessary  for  the  human  visual  search  performance 
testing. 

Human  Visual  Search  Performance  Evaluation 

In  order  to  evaluate  the  effects  of  the  model  on  human  visual  search  performance,  a  set  of  visual  stimuli 
were  created.  The  approach  was  to  systematically  manipulate  scale  and  orientation  characteristics  of  a 
background  image  to  create  target  elements  and  empirically  determine  the  perceptual  salience  of  these 
targets  with  human  observers.  These  data  provided  baseline  performance  information.  The  Neuronal 
Synchrony  Model  was  then  applied  to  the  imagery,  another  empirical  determination  of  the  perceptual 
salience  of  the  augmented  imagery  conducted,  and  a  comparison  of  the  effectiveness  of  the  model- 
augmented  imagery  with  that  of  the  baseline  imagery  carried  out.  The  creation  of  the  test  imagery  is 
described  briefly  in  the  next  section  and  In  detail  in  Marchak  (2002).  The  following  section  provides  the 
methods,  results,  and  analyses  of  the  visual  search  tasks. 


Test  Image  Creation 

To  test  human  performance  effectiveness  of  the  model,  seven  different  representative  background  images 
were  used.  IKONOS  imagery  of  desert,  forest,  jungle  and  mountains  as  well  as  SAR  imagery  of  desert, 
jungle  and  mountains  was  selected  (Figure  3).  These  images  were  chosen  to  provide  a  cross  section  of  both 
environments  and  sensor  types. 

A  key  factor  in  selection  of  the  particular  Gabor  algorithm  used  in  model  was  the  fact  that  it  needed  to 
support  both  texture  analyses  for  input  into  the  Neuronal  Synchrony  Model  and  texture  synthesis  for  the 
creation  of  the  test  images.  Test  image  backgrounds  were  created  by  first  analyzing  the  collection  of  images 
and  determining  their  Gabor  wavelet  coefficients.  Next,  these  measurements  served  as  the  input  to  the 
texture  synthesis  process  of  the  Gabor  algorithm  to  create  synthetic  background  images  based  on  the 
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original  imagery.  As  can  be  seen  by  comparison  of  Figures  3  and  4,  the  synthesized  backgrounds  captured 
the  fundamental  visual  characteristics  of  the  original  images. 


IKONOS  Forest  - 1  m  S  AR  Desert  -  24m  SAR  Mountains  -  200m 


SAR  Jungle  -  200m 

Figure  3.  Original  Imagery 
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Synthetic  IKONOS  Desert  Synthetic  IKONOS  Jungle  Synthetic  IKONOS  Mountains 


Synthetic  IKONOS  Forest  Synthetic  SAR  Desert  Synthetic  SAR  Mountains 


Synthetic  SAR  Jungle 

Figure  4.  Synthetic  Imagery 

The  Gabor  algorithm  was  also  used  to  create  controlled  perturbations  of  orientation  and  scale 
characteristics  of  the  synthetic  imagery.  Four  orientations  -  0,  45,  90,  and  135  degrees  -  and  four  scales  - 
64,  128,  256,  and  512  pixels  -  were  chosen  as  the  basic  permutation  parameters.  To  create  a  permutation, 
the  measured  Gabor  wavelet  coefficients  of  an  image  at  a  particular  orientation  or  scale  were  multiplied  by  a 
constant.  The  resulting  scaled  coefficients  were  then  used  to  synthesize  a  new  image  based  on  that 
perturbation.  Figure  5  shows  an  example  of  a  synthetic  background  and  the  same  image  with  an  orientation 
perturbation  at  90  degrees. 


,f  . . 

9 

.  %  *  ;  . 

:  .  ^*$4 

Original  90°  Orientation  Perturbation 

Figure  5.  Example  of  perturbed  imagery 
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Based  on  the  results  of  pilot  testing,  the  values  of  the  perturbations  were  manipulated  to  create  a  final  set  of 
3  levels  of  salience  for  each  perturbation  -  low,  medium  and  high  -  to  cover  the  perceptual  span  from 
subtle  to  obvious.  The  result  was  168  stimuli  made  up  of  3  levels  of  perceptual  salience  (low,  medium, 
high),  4  scales  (64,  128,  256,  512  pixels),  4  orientations  (0,  45,  90,  135  degrees)  and  7  backgrounds. 

For  each  background  type  and  perturbation,  a  test  image  needed  to  be  created  for  use  in  the  visual  search 
task.  In  this  approach,  a  36-pixel  diameter  circular  pattern  was  randomly  located  in  a  1000  x  1000  pixel 
background  image,  and  the  corresponding  pixels  from  an  identical  but  perturbed  image  were  place  in  the 
circle  (Figure  8).  The  targets  were  constrained  to  be  at  least  42  pixels  from  any  edge.  A  complete  set  of  test 
imagery  using  all  combinations  of  backgrounds,  scales,  orientations,  and  levels  of  perceptual  salience  was 
generated. 


Figure  8.  Final  test  stimulus 


Psychophysical  Testing  -  Synthetic  Imagery 

The  goal  of  this  task  was  to  conduct  a  visual  search  experiment  with  the  stimuli  created  above  in  order  to 
determine  baseline  values  of  observers’  visual  search  times  and  eye  movement  parameters  to  detect  targets 
embedded  in  imagery  that  has  not  been  augmented  with  the  output  of  the  Neuronal  Synchrony  Model.  The 
resulting  data  were  compared  with  those  obtained  using  augmented  imagery  to  determine  the  effectiveness 
of  this  approach  in  aiding  image  analysis. 

Subjects 

Twenty  subjects,  all  with  normal  or  corrected-to-normal  vision,  were  paid  for  their  volunteer  participation, 
with  ten  subjects  in  each  condition,  baseline  and  augmented.  The  Montana  State  University  Human 
Subjects  Committee  approved  experimental  procedures  and  protocols. 
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Apparatus 

A  BARCO  Personal  Calibrator  luminance-calibrated  monitor  was  used  to  ensure  accurate  stimulus 
presentation  across  subjects  and  data  collection  tasks.  A  Dell  Optiplex  GX1  running  E-Prime,  an 
experimental  presentation  and  data  collection  software,  was  used  for  image  presentation.  A  Dell  Optiplex 
GXi  was  used  to  collect  data  with  an  ISCAN  RK-426  remote  eye  tracking  system. 

Procedures 

The  E-Prime  experimental  presentation  software  was  configured  into  6  blocks  of  35  trials,  where  each  block 
contained  28  target  stimuli  and  7  catch  trials  with  no  target  present.  Subjects  were  first  presented  with  a 
series  of  informational  displays  that  described  the  task  and  provide  examples  of  hard-to-see  and  easy-to-see 
targets,  as  well  as  examples  of  the  different  background  types.  This  was  followed  by  a  series  of  35  practice 
trials  to  familiarize  them  with  the  procedures.  A  trial  began  with  a  fixation  point  in  the  center  of  the  screen. 
When  the  observer  pressed  a  key,  the  fixation  point  was  replaced  with  a  test  image,  the  search  timer  started 
and  eye  movement  tracking  began.  The  observer  had  up  to  30  seconds  to  indicate  detection  or  absence  of 
the  target  by  a  key  press,  which  recorded  the  visual  search  time  and  stopped  eye  movement  recording.  After 
a  short  delay  the  fixation  point  reappeared  and  the  subject  could  begin  the  next  trial.  The  practice  trials 
were  identical  to  the  experimental  trials  with  the  exception  that  the  data  were  not  analyzed. 

Half  the  subjects  performed  the  task  using  the  original  synthetic  imagery,  and  half  used  the  Neuronal 
Synchrony  Model  augmented  synthetic  imagery.  Figure  9  shows  an  example  of  the  original  imagery,  while 
Figure  10  shows  the  same  image  with  the  model  augmentation.  The  arrow,  not  shown  in  the  experiments, 
indicates  the  location  of  the  target. 
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performance  in  the  augmented  condition  was  superior,  this  difference  was  not  significant  (X2  -  1.345,  p  - 
0.131). 


Table  1 .  Synthetic  Imagery  Overall  Accuracy 


Accuracy  (Percent  Correct) 

Original  Imagery 

Augmented  Imagery 

42.6 

57.4 

When  the  data  are  examined  for  interactions  among  the  variables,  significant  differences  emerge.  Table  2 
shows  the  accuracy  for  each  background  type.  There  was  a  significant  overall  difference  in  accuracy  for 
background  type  (X2  =  25.85,  p  -  .000)  and  significant  interactions  of  image  condition  -  original  versus 
augmented  -  and  background  type.  Performance  was  better  with  the  augmented  imagery  for  the  IKONOS 
Forest  (X2  =  6.52,  p  =  .007),  IKONOS  Jungle  (X2  -  44.51,  p  -  .00),  IKONOS  Mountains  (X2  -  3.49,  p  « 
.038),  and  the  SAR  Jungle  (X2  =  24.61,  p  =  .000). 


Table  2.  Synthetic  Imagery  Accuracy  by  Background  Type 


Accuracy  (Percent  Correct) 

Background 

Original  Imagery 

Augmented  Imagery 

IKONOS  Desert 

42.1 

57.9 

IKONOS  Forest 

45.3 

54.7 

IKONOS  Jungle 

30.1 

69.9 

IKONOS  Mountains 

40.4 

59.6 

SAR  Desert 

44.1 

55.9 

SAR  Jungle 

47.6 

52.4 

SAR  Mountains 

44.0 

56.0 

Table  3  shows  the  accuracy  data  for  the  perturbations  in  scale  versus  the  perturbations  in  orientation.  There 
was  a  significant  difference  across  conditions,  with  augmentation  producing  the  greatest  benefit  for 
orientation  perturbations  (X2  =  15.81,  p  =  .000). 

Lastly,  examining  accuracy  level  as  a  function  of  degrees  of  perceptual  salience,  Table  4  shows  the  accuracy 
data  that  are  significantly  different  across  conditions.  The  greatest  benefit  of  augmentation  is  seen  for  the 
least  perceptually  salient  stimuli  (X2  ■  9.51,  p  =  .001),  and  the  medium  perceptually  salient  stimuli  (X2  = 
3.63,  p  =  .034). 
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Table  3.  Synthetic  Imagery  Accuracy  by  Perturbation  Type 


Accuracy  (Percent  Correct) 

Perturbation 

Original  Imagery 

Augmented  Imagery 

Scale 

50.7 

49.3 

Orientation 

46.2 

53.8 

Table  4.  Synthetic  Imagery  Accuracy  by  Perceptual  Salience  Level 


Accuracy  (Percent  Correct) 

Perceptual  Salience 

Original  Imagery 

Augmented  Imagery 

Low 

45.1 

54.9 

Medium 

50.8 

49.2 

High 

48.7 

51.3 

These  accuracy  data  show  that  the  Neuronal  Synchrony  Model  augmented  imagery  significantly  increase 
accuracy  of  performance  for  particular  backgrounds  (1KONOS  Jungle,  IKONOS  Mountains,  1KONOS 
Forest,  SAR  Jungle),  differences  in  orientation,  and  targets  of  low  and  medium  perceptual  salience. 

Turning  to  the  search  time  and  eye  movement  fixation  data,  a  multivariate  analysis  of  variance  (MANOVA) 
was  performed  on  the  correct  responses  with  background  type,  perturbation  type  (scale  and  orientation), 
and  level  of  perceptual  salience  as  factors.  Significant  levels  of  Wilks’  Lambda  were  found  between  the 
original  and  augmented  imagery  conditions  (F(2,27 09)  -  87.446,  p  *=  .000)  as  well  as  the  interaction  between 
background  type  and  condition  ( F(12,5418 )  =  2.876,  p=.00 1 ).  In  addition,  level  of  perceptual  salience  was 
also  significant  (F(3,5418)  -  3.103,  p  =  .015). 

Table  5  shows  the  means  for  overall  search  time  and  number  of  fixations  by  condition.  There  was  a 
significant  difference  in  search  time  (F(l,  2710)  =  151.880,  p  -  .000)  and  number  of  fixations  ( F(1 ,  2710)  = 
81.954,  p  ■  .000),  with  longer  search  times  and  a  greater  number  of  fixations  for  the  augmented  imagery 
versus  the  original  imagery. 


Table  5.  Synthetic  Imagery  Mean  Overall  Search  Time  and  Fixations 


Original  Imagery 

Augmented  Imagery 

Search  Time  (msec) 

2578.99 

4991,95 

Number  of  Fixations 

5.97 

15.85 
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Table  6  shows  mean  search  time  data  and  Table  7  shows  number  of  fixations  as  a  function  of  background 
type  for  both  conditions.  There  was  a  significant  effect  for  background  type  (F (6,2710)  =  3.059,  p  =  .006) 
and  a  significant  interaction  between  background  type  and  condition  (F(6,l)  =  4.549,  p  =  .000)  in  the  search 
time  data.  Subjects  spent  a  longer  time  searching  and  made  more  fixations  using  the  augmented  imagery 
than  using  the  original  imagery. 


Table  6.  Synthetic  Imagery  Mean  Search  Time  by  Background 


Search  Time  (msec) 

Background 

Original  Imagery 

Augmented  Imagery 

IKONOS  Desert 

3170.06 

4969.01 

IKONOS  Forest 

1169.06 

4948.10 

IKONOS  Jungle 

3650.94 

5148.35 

IKONOS  Mountains 

3382.11 

4763.59 

SAR  Desert 

2811.56 

4971 .07 

SAR  Jungle 

1288.18 

5136.07 

SAR  Mountains 

2581 .02 

5007.45 

Table  7.  Synthetic  Imagery  Mean  Number  of  Fixations  by  Background 


Number  of  Fixations 

Background 

Original  Imagery 

Augmented  Imagery 

IKONOS  Desert 

6.10 

16.54 

IKONOS  Forest 

5.41 

13.66 

IKONOS  Jungle 

5.87 

17.06 

IKONOS  Mountains 

6.63 

15.57 

SAR  Desert 

6.85 

15.18 

SAR  Jungle 

4.87 

16.39 

SAR  Mountains 

6.08 

16.56 

There  were  no  significant  differences  between  conditions  for  scale  and  orientation  perturbations.  Tables  8 
and  9  show  mean  search  time  and  mean  number  of  fixations  as  a  function  of  perturbations  in  scale  or 
orientation,  respectively. 
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Table  8.  Synthetic  Imagery  Mean  Search  Time  by  Perturbation  Type 


Search  Time  (msecs) 

Perturbation 

Original  Imagery 

Augmented  Imagery 

Scale 

2534.64 

2623.33 

Orientation 

4987.66 

4978.64 

Table  9.  Synthetic  Imagery  Mean  Number  of  Fixations  by  Perturbation  Type 


Number  of  Fixations 

Perturbation 

Original  Imagery 

Augmented  Imagery 

Scale 

5.73 

6.21 

Orientation 

16.30 

15.10 

Tables  10  and  11  show  mean  search  time  and  mean  number  of  fixations  as  a  function  of  level  of  perceptual 
salience. 


Table  1 0.  Synthetic  Imagery  Mean  Search  Time  by  Perceptual  Salience  Level 


Search  Time  (msec) 

Perceptual  Salience 

Original  Imagery 

Augmented  Imagery 

Low 

3199.87 

5316.55 

Medium 

2457.89 

4759.31 

High 

2079.21 

4873.59 

Table  11.  Synthetic  Imagery  Mean  Number  of  Fixations  by  Perceptual  Salience  Level 


Number  of  Fixations 

Perceptual  Salience 

Original  Imagery 

Augmented  Imagery 

Low 

5.79 

15.25 

Medium 

5.93 

16.14 

High 

6.19 

15.71 

As  the  data  above  show,  subjects  spent  more  time  searching  and  made  more  fixations  to  the  augmented 
imagery  than  to  the  original  imagery.  In  isolation,  this  finding  suggests  that  augmentation  by  the  Neuronal 


20 


VERIDICAL  RESEARCH  AND  DESIGN 


Synchrony  Model  causes  the  analyst  to  search  for  longer  periods  of  time  and  make  more  fixations  than 
searching  unaugmented  imagery.  However,  when  taken  together  with  the  accuracy  data,  a  different 
interpretation  may  be  reached. 

Examination  of  the  overall  accuracy  data  showed  that  this  particular  search  task  was  extremely  difficult,  with 
subjects  reaching  an  average  accuracy  of  just  over  50%  across  both  conditions.  In  the  original  imagery 
conditions,  the  shorter  search  times  and  fewer  fixations  could  be  attributed  to  subjects  terminating  their 
search  early  because  the  overall  homogeneity  of  the  stimuli  resulted  in  false  alarms  regarding  target 
detection.  While  there  were  catch  trials  with  no  target  present,  no  mechanism  was  available,  for  example,  by 
marking  the  target  location,  to  verify  that  when  subjects  indicated  detection  that  they  had  actually  found 
the  actual  target. 

In  the  augmented  imagery  condition,  the  highlighting  altered  the  homogeneous  nature  of  the  stimuli,  thus 
both  serving  to  better  indicate  true  targets  and  reduce  the  potential  for  falsely  identifying  targets.  Subjects 
searched  longer  and  made  more  fixations  in  the  highlighted  condition  because,  at  least  for  certain 
background  types,  the  highlighting  permitted  a  more  effective  search  and  resulted  in  less  frustration  than  in 
the  original  image  condition.  Informal  feedback  from  the  subjects  during  post-experiment  debriefing 
confirms  this  explanation,  as  those  in  the  original  imagery  condition  indicated  that  they  often  noticed 
features  of  the  background  that  they  thought  might  be  targets  and  that  after  an  initial  search,  they  would 
indicate  a  detection  if  they  noticed  almost  any  feature  that  differed  from  the  background.  Subjects  in  the 
augmented  imagery  conditions  commented  that  on  some  trials  the  target  would  apop-out”  of  the 
highlighted  background,  and  on  other  trials,  searching  the  areas  around  the  highlighting  often  led  to 
detecting  targets  that  were  otherwise  difficult  to  see. 

The  overall  findings  of  this  experiment  suggest  that  imagery  augmented  by  the  Neuronal  Synchrony  Model 
results  in  a  higher  level  of  detection  accuracy  and  an  increase  in  the  effectiveness  of  the  search  process  by 
increasing  the  amount  of  time  spent  successfully  searching  for  targets  and  reducing  the  tendency  to 
terminate  a  search  early  due  to  lack  of  feedback.  The  next  section  describes  refinements  made  to  the  model 
as  a  result  of  this  experiment  as  well  as  a  function  of  applying  the  model  to  actual  imagery.  These 
refinements  are  then  applied  to  overhead  imagery  and  the  results  evaluated  with  human  observers. 
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Task  3  -  Model  Refinement 

The  Neuronal  Synchrony  Model  developed  in  Task  1  increased  the  accuracy  and  success  of  visual  searches 
using  synthetic  imagery*  A  further  test  of  the  model’s  effectiveness  is  how  well  it  performs  on  actual 
overhead  imagery  of  natural  scenes*  The  goal  of  this  task  was  to  apply  the  model  to  representative  overhead 
imagery  and  refine  it  to  increase  its  performance  in  highlighting  areas  of  perceptual  salience 

Model  Application  To  Extended  Imagery  Set 

The  model  developed  in  developed  in  Task  1  was  applied  to  the  set  of  overhead  imagery  shown  in  Figure 
11.  Imagery  types  included  data  from  aerial  imagery,  IKONOS,  QuickBird,  Japanese  Earth  Resources 
Satellite  0ERS-1),  SAR,  and  SPOT. 
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Figure  1 1 .  Imagery  Set 
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QB  Kabul  244  cm  QB  Toledo  lm  QB  Trapini  SAR  1m  ABQ 


SAR  lm  Hangars  SAR  lm  Isleta  SAR  lm  Licong  SAR  3m  China  Lake 


SAR  Unknown  SPOT  10m  PA1  SPOT  10m  PA2  SPOT  10m  PA3 


Figure  1 1  (continued).  Imagery  Set 

Unlike  the  homogenous,  synthetic  imagery,  actual  overhead  imagery  contains  large  variations  in 
background  structure.  While  the  basic  elements  of  Gabor  filtering,  facilitation  and  inhibition  calculation 
functioned  well,  the  particular  implementation  of  neural  oscillator  was  unsuited  for  this  purpose.  This  is 
due  to  the  fact  that  the  original  LEGION  implementation  is  extremely  sensitive  to  noise  in  the  image 
because  of  an  oversimplified  local  coupling  scheme  (Chen,  Wang,  &  Liu,  2000).  To  address  this 
shortcoming,  an  extended  LEGION  model  that  contains  a  weight  adaptation  scheme  and  a  logarithmic 
grouping  rule  in  place  of  the  summation  and  maximization  rules  used  in  Terman  and  Wang  (1995)  was 
employed.  The  next  section  describes  the  changes  and  the  implementation  of  the  model 
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Update  and  Refinement 

The  sections  below  outline  the  design  of  the  updated  and  refined  Neuronal  Synchrony  Model  and  then 
specify  the  implementation  of  the  resulting  changes. 

Model  Design 

The  LEGION  model  by  Wang  and  Terman  (1995)  and  described  in  (8)  is  updated  in  Chen,  et  al.  ((2000) 
to: 


Xy  =3xv-xl +2- yy+  ItJH(ptJ  -e)  +  St+p 


(12a) 


y'ij  =  £(r( 1  +  tanh(xy  /  J3))  -  ytJ 


(12b) 


where  H(*)  is  the  Heaviside  step  function,  which  is  defined  as  H(v)  «  1  if  v  >  0  and  H(v)  =  0  if  v  <  0,  The 
Heaviside  term  provides  a  mechanism  to  distinguish  between  major  objects  and  noisy  fragments,  with  a 
major  object  required  to  contain  at  least  one  oscillator  located  at  the  center  of  a  large,  homogenous  region. 
Noisy  fragments  do  not  contain  such  oscillators.  The  overall  coupling  term,  $u  in  (12)  is  then  defined  as: 


S^Sl-WMz-0:)  (13) 

In  the  previous  algorithm  (Wang  Terman,  1995),  a  summation  rule  is  used  to  summate  the  weights  of 
neighboring  active  oscillators  and  a  maximization  rule  chooses  the  maximal  one.  In  the  revised  algorithm 
(Chen  et  al,  2000),  an  alternative  grouping  rule,  S%  using  a  logarithmic  operation,  is  defined: 


( 


log 


(14) 


where  is  the  activity  of  oscillator  (m,n)  and  where  1^  and  1^  are  the  minimal  and 

maximal  intensities  across  the  entire  image.  The  remaining  terms  are  as  described  in  the  previous  section. 

To  facilitate  weight  adaptation,  fixed  connections  are  assumed  in  the  neural  oscillator  network.  For  a  given 
oscillator,  the  fixed  connectivity  specifies  a  group  of  neighboring  oscillators  connecting  to  it.  This  oscillator 
corresponds  to  one  pixel  in  an  image,  with  the  lateral  attributes  associated  with  it  measured  from  an 
ensemble  of  fixed  connections.  The  fixed  connection  weights  are  determined  directly  from  the  image.  For 
oscillator  (i,j),  the  weights  of  its  fixed  connection  from  oscillator  (k, I),  T^u,  are  defined  as  the  difference 
between  the  external  stimuli  received  by  ( i,j )  and  (k,l)  in  its  lateral  neighborhood,  N(i,j,R): 
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t rJd  -  I/a  hj  (15) 

hj  and  Iu  are  the  intensities  of  pixel  ( i,j )  and  pixel  (fc,l)>  respectively,  and  N(i,j,R)  is  defined  as 

N(i,  j;R)  =  {(k,l)\i-R<k<i+R,j-R<l<j  +  R,(k,l)*(i, j)}  (16) 

where  R  (R  >  1)  is  a  parameter  that  determines  the  size  of  the  lateral  neighborhood.  For  oscillator  ( i,j i),  the 
fixed  connections  exist  only  in  N(i,j',R)  and  TifM  =  -Tm  where  (k,Q  e  N(i,j,R)  (Chen  et  al.,  2000). 

An  oscillator  corresponding  to  a  pixel  located  near  the  center  of  a  homogenous  region  tends  to  have  a  high 
potential  and  thus  can  lead  an  oscillator  group.  For  oscillator  (i,j),  ju^l),  a2u(l),  /uIJ(Rt>),  and  a2g(Rp)  are 
calculated  using  the  mean,  and  variance,  o2#,  of  the  fixed  weights,  defined  as 


m»(R) 


T 

s(k,l)e.N(i,j-,R)  >j&_ 


(17) 


(T^-mAR))2 


\N(i,j;R)\ 


(18) 


Oscillator  (ij)  is  the  center  of  a  homogenous  region  if  and  only  if  |  ^(Rp)  -  #/l)  |  <  and  |  a2^Rp)  - 
a2tf(l)  |<  Tm  where  and  %  are  two  thresholds  to  reflect  homogeneity  and  Rp  (RP>1)  reflects  whether  an 
oscillator  is  within  a  homogenous  region.  A  large  value  of  Rp  results  in  lead  oscillators  generated  from  large 
homogeneous  regions. 

Model  Implementation 

An  updated  LEGION  oscillator  network  was  implemented  in  MATLAB  based  on  the  equations  in  Chen  et 
al  (2002)  described  above  and  as  implemented  in  a  C  program  developed  by  Wang  (2001).  The  program 
begins  by  setting  up  the  connection  weights*  and  then  selects  leader  oscillators  in  homogeneous  regions, 
based  on  the  values  of  TM?  Tc,  and  R^  The  results  are  evaluated  to  determine  if  a  region  exists  and  the 
regions  are  then  numbered  in  a  manner  similar  to  that  described  above. 

The  Network  Filters  developed  in  Task  1  were  applied  to  the  imagery  in  Figure  11.  The  results  were 
submitted  to  the  revised  LEGION  algorithm,  with  the  following  range  of  parameter  values:  Rp  (4-12),  T^ 
(0.1-2),  Ta  (1-2),  R  (1-2),  and  Wz  (45-90).  Each  value  was  subjectively  determined  by  examining  the 
resultant  image  output.  The  output  of  the  LEGION  algorithm  is  a  pixel  map  that  assigns  a  unique  number 
to  each  region  defined  in  the  input  image  by  the  correlated  oscillations.  The  resulting  region  values  were 
each  mapped  to  saturation  levels  of  a  yellow  hue  to  produce  highlighted  regions.  All  other  pixels  were 
assigned  to  black.  The  highlights  from  the  oscillator  output  were  then  overlaid  on  the  original  imagery  to 
indicate  perceptually  salient  regions.  Figure  12  shows  the  resultant  augmented  images. 
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Figure  12.  (continued)  Augment  Real  imagery 


Psychophysical  Testing  -  Overhead  Imagery 

In  Task  2,  the  effectiveness  of  the  Neuronal  Synchrony  Model  was  evaluated  using  synthetic  imagery  with 
controlled  perturbations.  Under  this  task,  a  similar  methodology  was  applied  to  evaluate  the  effectiveness  of 
the  model  in  aiding  human  search  performance  when  applied  to  real  overhead  imagery.  Unlike  the  first 
visual  search  task,  it  was  not  possible  to  define  specific  targets  for  detection  in  the  real  imagery.  In  order  to 
produce  a  quantifiable  measure  of  image  contours  with  some  relevance  to  actual  image  analysis,  imagery  was 
chosen  that  contained  roads  and  intersections.  The  subjects*  task  was  to  count  the  number  of  intersections 
in  an  image.  If  the  Neuronal  Synchrony  Model  successfully  highlights  perceptually  salient  contours,  subjects 
in  the  augmented  condition  should  perform  better  than  those  in  the  original  imagery  condition. 

Baseline  visual  search  and  eye  movement  data  were  collected  using  the  original  imagery  shown  in  Figure  1 1 
and  compared  to  performance  using  the  augmented  imagery  shown  in  Figure  12.  Details  of  the  methods 
and  findings  are  presented  below. 

Subjects 

Twenty  subjects,  all  with  normal  or  corrected-tomormal  vision,  were  paid  for  their  volunteer  participation, 
with  ten  subjects  in  each  condition,  baseline  and  augmented.  The  Montana  State  University  Human 
Subjects  Committee  approved  experimental  procedures  and  protocols. 
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Apparatus 

A  BARCO  Personal  Calibrator  luminance-calibrated  monitor  was  used  to  ensure  accurate  stimulus 
presentation  across  subjects  and  data  collection  tasks.  A  Dell  Optiplex  GX1  running  E-Prime,  an 
experimental  presentation  and  data  collection  software,  was  used  for  image  presentation.  A  Dell  Optiplex 
GXi  was  used  to  collect  data  with  an  1SCAN  RK-426  remote  eye  tracking  system. 

Procedures 

The  task  of  the  subjects  was  to  count  the  number  of  road  intersections  in  each  of  the  images  presented.  The 
E-Prime  experimental  presentation  software  was  configured  into  1  block  of  29  trials,  containing  the  24 
augmented  stimuli  and  5  catch  trials  with  no  intersections  present.  Subjects  were  first  presented  with  a 
series  of  informational  displays  that  describe  the  task  and  provided  examples  of  different  intersection  types, 
as  well  as  examples  of  the  images  with  roads  but  no  intersections.  This  was  followed  by  a  series  of  5  practice 
trials  to  familiarize  them  with  the  procedures.  A  trial  began  with  a  fixation  point  in  the  center  of  the  screen. 
When  the  observer  pressed  a  key,  the  fixation  point  was  replaced  with  a  test  image,  the  search  timer  started 
and  eye  movement  tracking  began.  The  observer  had  unlimited  time  to  count  the  number  of  intersections 
in  an  image.  When  the  subject  had  made  a  determination,  they  pressed  a  key.  This  recorded  the  visual 
search  time,  stopped  eye  movement  recording,  and  presented  a  screen  that  asked  “How  many  intersections 
were  in  the  image?”  Subjects  entered  their  response  with  a  numeric  keypad.  After  a  short  delay  the  fixation 
point  reappeared  and  the  subject  could  begin  the  next  trial.  The  practice  trials  were  identical  to  the 
experimental  trials  with  the  exception  that  the  data  were  not  analyzed. 

Half  the  subjects  performed  the  task  using  the  original  imagery,  and  half  used  the  Neuronal  Synchrony 
Model  augmented  imagery. 

Results  and  Analyses 

Accuracy,  visual  search  time,  and  number  of  fixations  were  measured  for  each  image  and  image  type.  Table 
12  presents  the  overall  accuracy  data  across  sensor.  Similar  to  the  synthetic  imagery,  the  task  of  accurately 
counting  intersections  in  real  imagery  is  difficult;  in  the  Original  Imagery  condition,  subjects  averaged 
about  48%  correct  across  all  sensor  types,  while  with  the  augmented  condition,  accuracy  increased  to  52%. 
While  performance  in  the  augmented  condition  was  superior,  this  difference  was  not  significant  (X2  -  0.62, 
1>  -  -243). 

Table  12.  Real  Imagery  Overall  Accuracy 


Accuracy  (Percent  Correct) 

Original  Imagery 

Augmented  Imagery 

47.8 

52.2 

Table  13  shows  the  accuracy  for  each  sensor  type.  The  original  imagery  faired  better  with  radar,  LANDSAT, 
and  SPOT  imagery,  while  subjects  performed  better  using  the  augmented  imagery  with  aerial,  IKON OS, 
and  SAR  imagery.  Only  the  IKONOS  imagery  showed  a  marginally  significant  difference  (X2  =  2.88,  p  = 
.06),  favoring  the  augmented  imagery. 
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Table  13*  Real  Imagery  Accuracy  by  Sensor  Type 


Accuracy  (Percent  Correct) 

■BB 

Original  Imagery 

Augmented  Imagery 

Aerial 

46.3 

53.7 

IKONOS 

44.0 

56.0 

LANDSAT 

57.8 

41.2 

RADAR 

57.1 

42.9 

SAR 

45.0 

55.0 

SPOT 

56.3 

43.8 

Turning  to  the  search  time  and  fixation  data.  Table  14  shows  the  overall  average  search  time  and  number  of 
fixations  as  a  function  of  condition,  which  are  significantly  different  at  the  p<  .10  level,  with  (F (1,568)  = 
2.81,  p  *=  .094)  and  ( F(l,568 )  =  2.83,  p  =  .093),  respectively. 


Table  14.  Real  Imagery  Overall  Mean  Search  Time  and  Fixation  Data 


Original  Imagery 

Augmented  Imagery 

Search  Time  (msec) 

2075.49 

1804.91 

Number  of  Fixations 

34.60 

30.74 

An  examination  of  search  time  and  number  of  fixations  as  a  function  of  sensor  and  condition,  as  shown  in 
Tables  15  and  16,  shows  that  mean  search  time,  with  exception  of  SAR  imagery,  and  mean  number  of 
fixations  were  lower  for  the  augmented  imagery  than  the  original  imagery. 


Table  15.  Real  Imagery  Mean  Search  Time  by  Sensor  Type 


Search  Time  (msec) 

Image  Type 

Original  Imagery 

Augmented  Imagery 

Aerial 

2463.55 

1985.26 

IKONOS 

1912.30 

1667.69 

LANDSAT 

1994.60 

1645.90 

RADAR 

2152.63 

1761.63 

SAR 

2036.70 

2137.03 

SPOT 

1893.17 

1631.93 
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Table  16.  Real  Imagery  Number  of  Fixations  by  Sensor  Type 


Number  of  Fixations 

Image  Type 

Original  Imagery 

Augmented  Imagery 

Aerial 

31.93 

23.78 

IKONOS 

27.45 

25.73 

LANDSAT 

42.20 

36.10 

RADAR 

38.73 

36.60 

SAR 

34.36 

31.57 

SPOT 

32.93 

30.63 

Overall,  the  results  with  real  imagery  show  that  while  there  are  no  significant  differences  in  accuracy,  with 
the  exception  of  IKONOS  imagery,  between  un-augmented  and  Neuronal  Synchrony  Model  augmented 
imagery,  both  mean  search  time  and  number  of  fixations  are  lower  for  the  augmented  imagery.  These 
finding  could  be  due  to  several  reasons.  First,  the  particular  measure  of  accuracy,  as  measured  by  the 
number  of  intersections  counted,  might  not  be  an  adequate  evaluation  of  the  effectiveness  of  the 
augmentation.  Since  accuracy  rates  were  low  in  both  conditions,  the  task  itself  may  not  have  taken 
advantage  of  the  capabilities  of  the  model.  Second,  since  the  model  w7as  applied  at  only  one  spatial  scale  and 
facilitation  calculated  for  only  identical  orientation  filters,  the  full  fidelity  of  the  augmentation  may  not 
have  been  realized.  Applying  the  model  across  all  spatial  scales  and  across  all  combination  of  filter  and 
image  orientations  should  result  in  better  performance,  as  background  noise  present  at  only  one  scale 
would  be  reduced.  Lastly,  better  selection  of  threshold  values  for  the  selection  of  neural  oscillators  based  on 
objective  measures  of  the  characteristics  of  the  imagery  could  increase  identification  of  additional  salient 
contours. 
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Conclusions 

The  primary  objective  of  this  effort  was  to  develop  a  neurophysiologically  based  model  that  characterizes  the 
perceptually  relevant  contours  of  interest  in  overhead  imagery  of  natural  scenes  in  a  manner  consistent  with 
human  performance,  utilizing  only  information  available  in  the  image*  The  goal  was  to  apply  the  model  to 
highlight  contours  of  potential  interest  in  a  manner  that  augments  the  analyst’s  own  visual  processing.  The 
output  of  this  work  was  a  prootofconcept  demonstration  for  enhancing  the  analyst’s  ability  to  perform 
searches  on  representative  imagery. 

The  results  showed  that  for  targets  of  low  to  medium  perceptual  saliency  in  homogeneous  backgrounds,  the 
Neuronal  Synchrony  Model  increase  the  accuracy  of  detecting  those  targets  and  enhances  the  ability  to 
search  longer  with  fewer  false  alarms.  When  the  task  involves  determining  the  number  of  intersections  in 
real  overhead  imagery,  depending  on  the  sensor  type,  augmentation  with  the  Neuronal  Synchrony  Model 
reduces  average  search  time  and  number  of  fixations. 

In  order  to  demonstrate  the  feasibility  of  this  approach  within  the  constraints  of  the  current  effort,  several 
simplifications  were  made  to  the  model  implementation,  include  processing  at  only  a  subset  of  the  spatial 
scales  and  determination  of  facilitation  at  only  primary  orientations.  Despite  these  simplifications,  the 
model  nonetheless  locates  the  majority  of  perceptually  relevant  contours  in  an  image,  and  does  so  in  a 
manner  superior  to  standard  edge  detection  routines.  Figure  13  shows  the  output  of  the  Neuronal 
Synchrony  Model  compared  to  that  of  a  Canny  edge  detector  (Canny,  1986)  for  an  example  image. 


Figure  13  (a).  Output  of  Canny  edge  detector  Figure  13  (b).  Output  Neuronal  Synchrony  Model 

As  can  be  seen,  the  Neuronal  Synchrony  model  ignores  much  of  the  distracting  background  edges, 
highlighting  primarily  the  perceptible  contours  of  interest. 

Overall,  the  Neuronal  Synchrony  Models  show?s  great  promise  in  the  ability  to  provide  image  analysts  with  a 
measure  of  perceptual  salience  that  can  guide  image  search.  Further  research  and  expansion  of  the  model 
will  enhance  both  its  performance  and  utility.  The  next  section  outlines  topics  for  further  investigation. 
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Suggestions  for  Future  Research 

As  shown  above,  the  effectiveness  of  the  Neuronal  Synchrony  Model  algorithm  proved  to  vary  with  the 
different  image  types  and  search  tasks.  Several  factors  can  be  addressed  to  increase  the  performance  and 
robustness  of  the  algorithm. 

First,  as  a  simplification,  the  facilitation  and  inhibition  algorithm  operated  at  only  one  of  the  four  spatial 
scales  of  the  Gabor  filter  outputs.  Using  information  from  all  four  spatial  scales  would  result  in  the  ability 
both  to  detect  relevant  contour  features  across  scales  as  well  as  to  reject  background  noise  that  may  be 
present  at  only  one  scale.  In  addition,  facilitation  was  only  calculated  for  orientations  of  the  same  value  as 
the  facilitation  filter.  That  is,  the  45°  facilitation  filter  was  only  applied  to  the  Gabor  output  calculated  at 
45°.  Calculating  the  preferred  angle  between  a  particular  filter  and  each  orientation,  thus  summing  energy 
that  occurs  across  all  filter  outputs,  could  capture  a  more  accurate  level  of  facilitation. 

Second,  the  revised  LEGION  method  of  modeling  neuronal  oscillation  implemented  on  the  real  imagery, 
based  on  the  work  of  Chen  et  al.,(2000),  incorporated  only  fixed  weights  to  specify  connections  between 
oscillators  in  the  network.  This  fixed  connectivity  specifies  a  group  of  neighboring  oscillators  connected  to 
an  individual  oscillator  and  is  determined  directly  from  the  image.  As  described  in  Chen  et  al.,  (2000), 
dynamic  weight  connections  can  also  be  employed.  These  dynamic  weights  capture  the  instantaneous 
relationship  between  two  oscillators  during  weight  adaptation,  providing  noise  removal  and  feature 
preservation.  Further,  the  particular  values  for  oscillator  threshold  variables  were  selected  manually  and  ad- 
hoc  for  each  individual  image.  Development  of  schemes  for  automatically  selecting  the  currently  manually 
selected  parameter  values  for  homogeneity  and  determination  of  whether  an  oscillator  is  in  a  homogeneous 
region  would  make  the  process  both  more  objective  and  more  effective. 

While  the  goal  of  the  this  effort  was  to  detect  open  contours  in  imagery,  the  model  on  which  the  algorithm 
is  based  (Yen  &  Finkel,  1998)  is  especially  effective  at  detecting  closed  contours,  which  have  a  higher  degree 
of  perceptual  salience.  It  is  possible  to  extend  and  expand  the  current  algorithmic  implementation  of  the 
model  to  optimize  the  ability  to  detect  perceptually  salient  features  composed  of  closed  contours,  such  as 
buildings  and  vehicles.  This  would  involve  investigating  the  optimal  implementation  of  the  horizontal  long- 
range  connections  used  in  the  model,  as  well  as  tuning  the  facilitation  and  inhibition  calculations. 

Lastly,  as  discussed  above,  the  method  of  highlighting  used  on  the  imagery  was  based  on  applying  a  single 
hue  and  intensity  to  all  background  types.  A  more  effective  methodology  would  be  to  select  hue  and 
intensity  values  based  on  the  contrast  and  color  of  the  image  to  be  augmented.  Investigation  of 
methodologies  for  adaptively  determining  the  optimal  highlighting  scheme  based  on  the  characteristics  of 
the  underlying  imagery  should  increase  human  performance. 
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